This article introduces the pyramid search approach using Agentic Knowledge Distillation to address the limitations of traditional RAG strategies in document ingestion.
The pyramid structure allows for multi-level retrieval, including atomic insights, concepts, abstracts, and recollections. This structure mimics a knowledge graph but uses natural language, making it more efficient for LLMs to interact with.
Knowledge Distillation Process:
- Conversion to Markdown: Documents are converted to Markdown for better token efficiency and processing.
- Atomic Insights Extraction: Each page is processed using a two-page sliding window to generate a list of insights in simple sentences.
- Concept Distillation: Higher-level concepts are identified from the insights to reduce noise and preserve essential information.
- Abstract Creation: An LLM writes a comprehensive abstract for each document, capturing dense information efficiently.
- Recollections/Memories: Critical information useful across all tasks is stored at the top of the pyramid.
This article explores how to boost the performance of small language models by using supervision from larger ones through knowledge distillation. The article provides a step-by-step guide on how to distill knowledge from a teacher model (LLama 2–70B) to a student model (Tiny-LLama) using unlabeled in-domain data and targeted prompting.
